COMPUTER-BASED  INSTRUCTIONAL  RESEARCH  AND  DEVELOPMENT  IN  THE  AIR  FORCE:  AN  OVERVIEW 

The  Air  Force  lias  been  imolsed  in  the  application  oi  computer  tcchnologs  to  training  lot  a  considerable  time.  One  ol  the  vers  calls  el  I  oils  was  the  computet -based 
training  capability  developed  lor  on-the-job  training  ol  operators  within  the  semi-amomatie  ground  environment  (SACiL)  system  in  the  mid  1950  s.  The  basic  notion  ol 
using  an  operational  computer  to  provide  on  the  job  training  was  later  extended  to  the  Air  I  oree  computer  directed  training  system  (CDIS)  This  system,  winch  *  soil 
in  use  provides  CAl  delivered  training  lor  a  variet>  ol  base  level  support  personnel 

Although  preliminarv  concept  slud.es  were  conducted  during  the  1960^  regarding  the  potential  ot  "automated’  insirueuon  and  the  use  ot  computers  in  naming,  it 
was  not  until  the  earlv  1970’s  that  the  Air  Force  began  a  long  range  research  and  development  thrust  in  computer-based  instruction  Initial  el  Ions  included  an 
assessment  of  the  application  of  CAl  at  that  time  and  demonstration  studies  ot  the  use  oi  CAl  to  teach  Air  Force  technical  courses  Other  related  investigations  ex¬ 
plored  the  feasibility  ot  computer  controlled  adaptive  testing  models,  artificial  intelligence  based  troubleshooting  and  instructional  management  The  major  Air  Force 
CB1  research  and  development  effort  during  the  1970’s  was  the  Advanced  Instructional  System  (A1S)  Fhis  $ttcm  was  designed  to  provide  the  capability  to  administer 
and  manage  individualized  instruction  on  a  large  scale  A  major  state-of-the-art  advancement  was  a  software  capability  which  provided  a  full  range  ot  C  B1  functions 
including  course  development  and  presentation,  resource  allocation  and  scheduling,  and  instructional  management  The  papers  within  this  section  are  representative  ot 
the  current  directions  in  Air  Force  CB1  research  and  dev  elopment 

The  views,  opinions,  and' or  findings  contained  in  this  report  are  those  of  the  author  and  should  not  be  construed  as  an  olticial  Department  ol  the  Air  fotce  or 
Department  ot  Defense  position,  policy';  or  decision,  unless  so  designated  by  other  official  documentation 
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DESK  TOP  TRAINER: 

TRANSFER  OF  TRAINING  OF  AN  AIRCREW  PROCEDURAL  TASK 


David  L.  Pohlman  and  Bernell  J.  Edwards 
USAF  Human  Resources  Laboratory 


This  paper  describes  a  trial-of-concept  experiment  conducted  to  evaluate  the  utility  ol  a  microcomputer  graphics-based 
desk  top  trainer  (DTT)  for  aircrew  training  purposes.  Air  Force  instructor  and  student  pilots  were  trained  to  program 
the  computer  controlled,  air-to-ground  weapons,  on  an  F- 1 6  aircraft .  An  experimental  group  (n  =  20)  was  trained  in  the 
procedure  using  a  microcomputerized  desk  top  trainer  while  a  control  group  (n  =  20)  was  trained  using  a  sell- 
instructional  illustrated  text.  Transfer  of  training  effects  were  assessed  using  tasks  on  the  actual  aircraft  system  as  the 
criterion,  with  time  and  number  of  errors  as  the  dependent  measures  The  experimental  group  was  able  to  complete 
criterion’ tasks  in  significantly  less  time  and  with  significantly  fewer  errors,  than  the  control  group,  A  transfer  ef¬ 
fectiveness  measure  (TEM),  computed  for  both  dependent  measures,  was  found  to  be  significant  for  the  experimental 
group  Results  support  the  application  of  microcomputer  graphics  systems  as  low-cost  alternative  training  systems  for 
the  procedural  aspects  of  aircrew  training 


The  cost  of  training  Air  Force  aircrews  continues  to  rise.  Energy 
inflation,  developmental  and  operational  costs  now  make  flight 
simulators  and  training  devices  nearly  as  expensive  as  the  aircraft 
themselves.  Although  simulators  are  in  widespread  use  within  the 
Air  Force,  there  is  an  urgent  need  for  less  costly  training  alter¬ 
natives. 

The  transfer  of  learning  from  a  simulated,  training,  environment  to 
an  operational  environment  is  a  major  issue.  Questions  relating  to 
task  fidelity  (e.g.  how  closely  a  training  system  must  approximate  an 
actual  system  in  terms  of  performance  functions,  appearance,  etc.) 
are  at  issue  whenever  new  technology  is  considered.  With  cost  as  a 
primary  consideration,  one  approach  to  improving  training  is  to 
optimize  efficiency  during  a  pre-simulator  phase  by  exploiting  least- 
cost  technologies. 


The  views,  opinions,  and/or  findings  contained  in  this  report  are  those  of 
the  authors  and  should  not  be  construed  as  an  official  Department  of  the  Air 
Force  or  Department  of  Defense  position,  policy,  or  decision,  unless  so 
designated  by  other  official  documentation 
Requests  for  reprints  should  be  sent  to  LTC  David  L.  Pohlman, 
AFHRL/1D,  Lowry  AFB,  CO  80230 


Related  Literature 

Little  is  known  about  the  potential  transfer  of  learning  benefits  of 
microcomputerized  trainers  for  aircrew  tasks  as  measured  on  actual 
aircraft  equipment.  The  scarcity  of  empirical  evidence  is  particularly 
troublesome  in  view  of  the  widespread  interest  in  the  applying  of 
microcomputer-based  systems,  as  low-cost  devices,  to  aircrew 
training. 

The  relative  effectiveness  of  computer  assisted  instruction  (CAl) 
versus  illustrated  program  text  was  examined  in  this  study.  The  text 
version  of  training  provided  a  convenient,  low  cost,  alternative 
media  representative  of  much  of  the  material  presently  used  in 
aircrew  academic  settings.  Related  research  supports  the  superiority 
of  CAl  over  text  or  other  training  media.  (Deignan  et  al,  1980; 
Robinson,  Tomblin  and  Houston,  1981;  Buck,  1982;  and  Fisher, 
1982).  Subjects  in  those  studies,  however,  were  dissimilar  to  those 
used  in  the  present  investigation  (i.e.  neither  college  graduates  nor 
pilots),  and  therefore  precluded  generalizability  of  results. 

The  objective  of  this  study,  therefore,  was  to  evaluate  transfer  of 
learning  to  actual  aircraft  equipment  from  training  using  a 
microcomputer-assisted  interactive  graphics  trainer  in  comparison 
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lo  an  illuslraled  programmed  text  containing  equivalent  training 

content,  using  subjects  who  would  typically  perform  the  criterion 
tasks 


METHOD 


Experimental  Design 

A  modified  post-test  only  control  group  design  (Campbell  and 
Manley,  1971)  was  used  to  compare  group  performance  on  two 
dependent  measures:  mean  time  to  task  completion  and  mean  error 
rate  per  task.  In  addition,  a  general  linear  model  employing  training 
time  as  a  covariate  (Pennell,  note  1),  was  used  to  assess  transfer  of 
training  effects.  The  model  was: 

Y'  =  a  +  b,G  +  b2X  +  b,(GX)  (1) 

Where  Y'  =  Predicted  value  of  the  dependent  variable, 
a  =  intercept, 

G  =  design  or  dummy  variable 

expressing  group  membership, 

X  =  covariate, 
bi-bj  -  Regression  weights, 

GX  =  Interaction. 

If  b,  is  significant,  the  within  group  regressions  are  not 
homogenous.  Then  it  can  be  argued  that  a  transfer  effectiveness 
measure  (TEM)  should  be  a  function  of  the  separate,  non- 
homogenous  regression  weights.  However,  when  b3  is  not  significant 
(i.e.  the  regression  is  homogenous),  a  natural  TEM  is- 

TEM  =  \-  Y\_  (2) 

Y'c 

(assuming  bj  is  significant).  TEM  varies  between  0  and  1  and 
measures  the  superiority  of  experimental  training  for  a  fixed  amount 
of  training  time. 

This  methodology  was  used  rather  than  the  transfer  effectiveness 
ratio  (TER)  reported  by  Roscoe  (1971,  1972)  and  Payne  (1982) 
because  these  models  assume  no  pre-training  of  the  control  group' 

In  Hoopoe's  treatment,  this  assumption  is  used  as  a  basis  for  the 
TEH  computation.  Roscoe  uses  the  value  for  the  control  group’s 
criterion  score  as  the  baseline  for  the  TER  evaluation.  Since  the 
present  experiment  was  planned  with  the  control  group  receiving  a 
version  of  training,  neither  1  ER  nor  the  percent  of  transfer  of 
learning  discussed  by  Payne  was  appropriate  for  the  assessment  of 
transfer  of  training  in  the  present  study. 


Figure  1 . 


Figure  2 


Subjects 

Instructor  pilots  (n  =  20)  and  student  pilots  (n  —  20)  assigned  to 
undergraduate  pilot  training  at  Williams  AFB  participated  as 
subjects.  None  had  knowledge  of  the  F-16  aircraft  stores 
management  system.  Subjects  were  randomly  assigned  to  groups. 


Experimental  Tasks 

The  task  was  programming  the  air-to-ground  weapons  profile 
mode  of  the  F-16  stores  management  set.  The  stores  management  set 
is  a  computer  program  within  the  aircraft  Tire  control  computer 
which  permits  the  pilot  to  preset  various  release  parameters 
associated  with  air-to-ground  weapons  prior  to  release  of  the 
weapons  during  a  mission.  The  pilot  controls  the  system  by  the 
stores  control  panel  (SOP)  which  is  about  five  inches  square  and 
consists  of  push  button  controls  surrounding  an  illuminated 
alphanumeric  display  (see  Figures  I  and  2).  By  pushing  buttons  in 
the  proper  sequence,  the  pilot  can  select,  inventory,  or  modify 
delivery  parameters  for  up  to  five  separate  weapons,  The  criterion 
tasks  used  in  this  experiment  consisted  of  programming  each  oT  five 
weapons  according  to  specific  release  parameters. 


Performance  Measures 

Time  on  task  and  number  of  input  errors  were  the  performance 
measures.  The  criterion  test  consisted  of  five,  5"  x  1"  cards,  each 
containing  a  different  specification  for  a  weapon  delivery  profile. 
Each  task  description  specified  weapon  type,  specific  release 
parameters,  and  required  subjects  to  program  the  system  by  control 
inputs  (i.e.  button  pushes)  on  the  stores  control  panel  in  a  cockpit 
environment.  None  of  the  (asks  in  the  criterion  test  were  identical  eo 
any  examples  or  practice  items  used  in  either  version  of  ihe  training 
materials. 
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DESK  TOP  TRAINER 


Training  Materials 

The  display  functions  of  the  stores  control  panel  were  modeled 
graphically  on  the  desk  top  trainer.  A  computer  program  was 
developed  to  simulate  all  the  relevant  functions  of  the  aircraft 
system  for  the  air-to-ground  weapons  delivery  profile  mode.  A  self- 
instructional  training  sequence  was  developed  so  that  subjects  could 
recei\ e  all  pertinent  information,  and  interact  as  necessary  with  the 
system  to  learn  proper  task  procedures.  Programmers  used  the  F-16 
stores  management  set  software  from  the  Advanced  Simulator  for 
Pilot  Training  as  a  reference  in  developing  experimental  software 
for  the  desk  top  trainer.  Prior  to  the  conduct  of  the  study  the  system 
was  operated  by  experienced  F-16  pilots  as  a  validation  of  technical 
content. 

An  illustrated,  programmed  text  was  also  developed  containing 
task  information  equivalent  to  that  in  the  desk  top  trainer  version. 
The  illustrated  text  was  prepared  using  the  F-16  Avionics  manual  as 
a  content  guide.  The  text  and  illustrated  material  corresponded 
closely  with  the  computerized  instruction,  but  without  the  manual 
interactivity  feature.  The  text  was  illustrated  to  provide  clear  visual 
representations  of  each  step  or  procedure,  exact  control  inputs  to  be 
made,  and  system  reactions  to  each  control  input.  The  illustrated 
text  was  reviewed  and  edited  by  content  experts  and  tried  out  by 
several  pilots.  The  approximate  time  required  to  complete  in¬ 
struction  with  each  version  of  training  was  verified  prior  to  data 
collection  as  a  control  for  floor  effects. 


Apparatus 

The  desk  top  trainer  (Figures  3  and  4)  was  based  upon  a  64K-byte, 
random  access  memory,  microcomputer.  The  system  used  an  S100 
buss  and  dual  mini-floppy  disk  drive.  The  primary  display  mode  was 
a  color  graphics  display.  Programming  and  alphanumeric  texts  were 
provided  using  a  standard  terminal.  Interface  with  the  graphics 
display  was  achieved  through  a  touch  panel  installed  upon  the 
graphics  display  and  integrated  with  the  computer. 

In  the  stores  control  panel  configuration,  the  graphic  represen¬ 
tation  of  the  panel  was  about  40  percent  larger  than  the  actual 
aircraft  equipment.  The  training  text  was  displayed  on  the  terminal 
cathode  ray  tube. 

A  functional  spare  F-16  stores  control  panel  was  used  for  criterion 
testing.  This  panel  was  contained  in  the  F-16  cockpit  of  the  Ad¬ 
vanced  Simulator  for  Pilot  Training  and  permitted  capture  of 
performance  data.  Because  the  system  was,  in  fact,  actual  aircraft 
equipment  and  functioned  precisely  as  does  the  aircraft  system 
stores  control  system,  the  assumption  of  equivalence  (i.e.  high 


Figure  3 


Figure  4 


fidelity),  between  the  simulator  environment  and  the  aircraft  ap¬ 
peared  reasonable  in  the  present  experiment. 


Procedures 

Subject  scheduling  for  experimental  training  and  randomized 
assignment  to  groups  were  predetermined.  Each  subject  was  given  a 
standard  text-inbriefing  and  asked  to  provide  some  basic 
demographic  data,  regardless  of  group  assignment. 

Subjects  in  the  experimental  group  were  seated  at  the  desk  top 
trainer  and  given  a  brief  introduction  which  included  operating 
instructions  after  which  the  experimental  training  proceded.  At  the 
end  of  training,  subjects  were  allowed  to  ask  questions  about  lesson 
content. 

The  control  group  received  the  self-instructional  illustrated  text 
and  proceeded  with  the  programmed  instruction.  At  the  conclusion 
of  the  instruction,  the  subjects  were  allowed  to  ask  questions  about 
the  procedure  and  clarify  points  in  the  lesson.  The  time  from  start  to 
completion  of  instruction  was  timed  for  all  subjects  in  both  groups. 

Following  completion  of  instruction,  all  subjects  were  escorted 
directly  to  the  simulated  F-16  cockpit  for  criterion  testing  with  each 
receiving  the  same  tasks.  They  were  shown  the  cockpit,  seated,  and 
briefed  on  the  stores  control  panel.  Subjects  were  told  to  execute 
each  of  the  five  tasks  contained  on  the  task  cards  one  at  a  time,  as 
quickly  as  possible,  but  with  as  few  errors  as  possible.  The  time  to 
complete  each  task  and  the  total  elapsed  time  for  the  entire  test  were 
recorded.  The  order  of  the  five  tasks  was  the  same  for  all  subjects. 
Information  displayed  on  the  stores  control  panel  was  the  same  at 
the  start  of  each  task  for  all  subjects. 


Error  Analysis 

In  addition  to  time  measures,  the  number  of  control  inputs  made 
by  each  subject  for  each  task  was  recorded.  The  programming  of 
each  task  required  a  specific  sequence  of  pushed  button  inputs. 
Because  each  task  was  initiated  with  the  same  information  displayed 
on  the  stores  control  panel,  there  was  a  most  direct  or  shortest 
“path”  to  correctly  programming  each  profile,  (i.e.,  the  minimum 
number  of  button  inputs  to  affect  the  correct  alphanumeric  display). 
Error  inputs,  therefore,  could  be  tabulated  simply  by  subtracting  the 
number  of  inputs  required  by  the  subject  from  the  number  required 
for  errorless  execution  of  the  task.  A  list  of  the  minimum  number  of 
inputs  per  task  is  shown  in  Table  l. 
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Table  1  Shortest  (Most  Correct)  Path  to  Criterion 

Task  Number  12  3 

Buttons  Required  13  14  3 


RESULTS 

The  comparison  of  group  performance  on  the  criterion  test  is 
summarized  in  Figure  5.  The  experimental  group  mean  completion 
time,  per  task,  for  the  five  tasks  in  the  criterion  test,  was  38.9 
seconds  in  comparison  to  47.5  seconds  for  the  control  group 
(standard  error  of  mean  =  5.2  sec).  An  alpha  level  of  .05  was 
selected.  The  F- value  for  this  comparison  was  6.82  (ds  =  1 ,  39;  p  = 
.013).  The  mean  errors  (excess  button  pushes)  made  by  the  ex¬ 
perimental  group  was  .93  per  task  compared  to  3.3  per  task  for  the 
control  group  (standard  error  of  mean  =  .98).  The  obtained  F-value 
for  this  comparison  was  14.17  (ds=  1 ,  39;  p  <.00i). 


EXPERIMENTAL  RESULTS 


N=20  Na2Q 


The  linear  model  (1)  was  evaluated  for  both  dependent  measures 
(i.e.,  time  to  task  completion  and  errors).  In  both  cases  the  within 
group  regressions  were  homogenous  and  the  group  effects  (b,)  were 
significant.  Using  equation  (2),  therefore,  the  TEM  for  time  and 
errors,  both  statistically  significant,  was  .183  and  .182  respectively. 


DISCUSSION 

This  experiment  was  intended  primarily  as  a  trial-of-concept  of 
microcomputer  technology  for  aircrew  training.  The  purpose  was  to 
assess  the  effectiveness  of  a  desk  top  trainer  as  a  vehicle  to  train 
cockpit  procedural  tasks.  As  a  part  of  this  assessment,  experimental 
training  was  compared  to  a  version  of  instruction  often  used  in 
current  academic  training  environments.  Comparision  of  criterion 


performance  differences  between  the  two  groups,  and  the  dif¬ 
ferential  transfer  of  training  effects  to  criterion  performance  on 
aircraft  equipment,  demonstrated  the  superiority  of  the  desk  top 
trainer  instruction.  The  desk  top  trainer  was  more  effective  than  the 
text  version,  presumeably  because  the  system  gave  subjects  the 
opportunity  to  actually  engage  in  the  tasks  in  a  manner  nearly 
identical  to  that  of  the  cockpit  environment. 

In  order  to  fully  test  the  contribution  of  the  interactivity  element, 
however,  an  empirical  comparison  would  be  required  between 
di ft erem  versions  of  the  CAL  one  with  and  one  without  interactivity 
te.g.  Avner,  Moore,  &  Smith,  1980).  The  present  study  provided  a 
promising  initial  examination  of  an  application  of  a  “micro"  for  an 
aircrew  procedural  task.  The  results  also  suggested  the  ap^ 
propriateness  of  a  formal  comparison  of  a  desk  top  trainer  against 
aircraft  equipment  itself.  The  results  of  such  a  test  would  provide  a 
definitive  baseline  to  establish  the  actual  cost  avoidance  potential  of 
the  desk  top  trainer  for  an  operational  training  context.  Such  an 
et  Ion  was  clearly  outside  the  scope  of  the  present  data. 

it  seems  reasonably  certain  that  microcomputers  can  be  effectively 
applied  to  a  variety  of  aircrew  tasks  or  part-tasks.  But  within  this 
generalization  there  remains  a  great  deal  to  be  learned  about  op¬ 
timizing  technological  applications.  Important  issues  related  to 
courseware  development,  cost  effectiveness,  software  stan¬ 
dardization  and  management,  effectiveness  evaluation,  and  others 
will  require  systematic  investigation  before  instructional  technology 
can  be  applied  with  confidence. 


REFERENCE  NOTE 

l.  Pennell,  R.  J  Personal  communication  on  the  appropriate  statistic  for 
transfer  effectiveness  ratio  when  both  experimental  and  control  groups 
receive  training,  May  1983. 
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